Pull Request Overview

Introduces a new streaming-based serialization architecture replacing vector-returning functions with pluggable output_stream abstractions (memory, file, chunked) and adds higher-level serializer / chunk_serializer utilities. Refactors flatbuffer construction logic into flatbuffer_utils, moves type mapping out of utils, and adds size‑estimation helpers for preallocation.

• Added output_stream interface with memory, file, and chunked implementations plus new (chunk_)serializer classes
• Refactored flatbuffer and body serialization into modular helpers; functions now write directly to streams
• Added size estimation utilities (calculate_*_message_size / calculate_total_serialized_size) and extensive new tests

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 20 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

The implementation is neat, I have just a few remarks regarding the architecture and API.

API

We want to be able to write something like:

std::ofstream out("my_file");
auto serializer ser(out);
ser << my_schema << my_array << my_list_of_batches << end_of_record;

Serializer

This means that:

• The serializer must accept any kind of streams, including the standard ones (see next section for more detail)
• The serializer should store an internal state (Waiting for a schema before accepting array ror record batches for instance, etc) so that its constructor does not need to accept a record_batch

The serializer methods names should reflect the stream methods names (i.e. append is actually write); also it should contain all the logic specific to sparrow, like adding padding.

Streams

The hierarchy of streams is actually a hierarchy of stream adaptor. I think this hierarchy can be removed, and some concepts can be provided instead, to help the implementation of the serializer:

template <class T>
concept output_stream = requires(T& t, const char* str)
{
    t.write(str, size_t(0));
    t.flush();
};

template <class T>
concept reservable_output_stream = output_stream<T> && requires(T& t)
{
    t.reserve(size_t(0));
};

If you need a layer between the streams and the serializer to adapt the signatures (because streams accept const char* while you serialize into std::span<uint8_t>, a stream_adapter class can be used to avoid repeating the cast everywhere. This layer should not contain additional logic like add_padding, which should live in the serializer.

Markers

The markers (for indicating the end of a record_batch list for instance) should follow the same pattern as std:endl for instance: a function accepting and returning a serializer.